To compress the data size of a moving image, encoding of the moving image is performed. As an example of a system of encoding a moving image, the high efficiency video coding (HEVC) having more than twice the encoding efficiency of H.264/AVC or H.264/AVC has been standardized.
When a moving image is encoded, one encoding mode is selected from among a plurality of encoding modes in order to enhance the encoding efficiency. An example of the encoding modes includes a block size with which an image that forms a frame of a moving image is divided. For example, in the case of the HEVC, five types of block sizes: 4×4, 8×8, 16×16, 32×32, and 64×64 can be selected, and a block size having the highest encoding efficiency is selected from the five types of block sizes.
When an encoding mode is selected as described above, an encoding cost that evaluates the cost of the encoding mode from both sides of the image quality and the compression rate before/after encoding is obtained, as an example. As an example, the encoding cost is calculated for each encoding mode using the following formula (1). “Encoding error Dist” in the following formula (1) is a sum of squares of a difference between a decoded image (Recon[i]) and an original image (Org[i]). Further, “information amount bit” in the following formula (1) is an information amount generated when variable length encoding is applied to an orthogonal transformation coefficient obtained by execution of orthogonal transformation and quantization to a difference image between the prediction image and the original image. Further, “λ” in the following formula (1) is the Lagrange multiplier.An encoding cost=Dist (encoding error)+λ*Bit (information amount)  (1)
Specifically describing the above, after execution of orthogonal transformation and quantization to a difference image between a prediction image and an original image, a decoded image is obtained by execution of inverse quantization and inverse orthogonal transformation, and then an encoding error that is an error between the decoded image and an input image is calculated. Further, variable length encoding is executed to a quantization coefficient obtained by execution of orthogonal transformation and quantization to the difference image, and an information amount generated in the variable length encoding is estimated. Then, an encoding cost is calculated from the encoding error and the information amount calculated in advance. After the encoding cost is calculated for each encoding mode, an encoding mode having the lowest encoding cost is selected as an optimum encoding mode.
Patent Literature 1: Japanese National Publication of International Patent Application No. 2005-527159
Patent Literature 2: Japanese Laid-open Patent Publication No. 2006-262004
However, in the above technology, the operation amount of the encoding cost becomes huge.
That is, in the case of the above technology, the encoding error can be calculated only after the decoded image is obtained by execution of the orthogonal transformation, the quantization, the inverse quantization, and the inverse orthogonal transformation, with respect to the difference image between the prediction image and the original image. Further, the information amount is not estimated unless the variable length encoding is executed after the execution of the orthogonal transformation and the quantization to the difference image. As described above, in the above technology, the encoding cost is not calculated unless the orthogonal transformation, the quantization, the inverse quantization, the inverse orthogonal transformation, and the variable length encoding are executed. Thus, a huge operation amount occurs in calculation of the encoding cost.